In [1]:
# This is the input block -- a full bone Python Shell
print('Look: I will be shown on output block')
Useful tricks in IPython notebook¶
In [2]:
import pprint
from IPython.core.display import HTML
In [3]:
HTML('Logo of Initium Lab: <img src="%s">' % 'http://initiumlab.com/favicon-32x32.png')
Out[3]:
Logo of Initium Lab: 
In [4]:
# Display any HTML easily
my_html = '''
I'm going to show you:
<ul>
<li> PyReadability </li>
<li> PyQuery </li>
<li> ... </li>
</ul>
'''
HTML(my_html)
Out[4]:
A small hack to allow longer output area¶
In [5]:
%%javascript
//IPython.OutputArea.auto_scroll_threshold = 9999;
IPython.OutputArea.prototype._should_scroll = function(){return false;}
Why Scraping?¶
In [6]:
# I'm going to insert some slides here
from IPython.core.display import Image
Related works @ Initium Media¶
Hong Kong Legislative Council:
https://theinitium.com/article/20150812-hongkong-legcoanalysis/
http://legco.initiumlab.com/matrix
Hong Kong District Council Election:
https://theinitium.com/project/20151012-hk-district-council-elections/
https://theinitium.com/project/20151019-hk-district-council-elections-2/
https://theinitium.com/project/20151029-hk-district-council-elections-3/
In [7]:
Image('assets/venn-skillset.png')
Out[7]:
In [8]:
Image('assets/workflow-highlight-data-collection.png')
Out[8]:
About collecting data¶
- Open Data (easy to find; machine readable; free to use) -- Good.
- Public data (but not "open data") -- Needs scraping
- Private data -- an issue of manpower
- Inputing
- Labeling
In [9]:
print('screenshot from: https://cerg1.ugc.edu.hk/cergprod/scrrm00541.jsp')
Image('assets/rgc-official-site.png')
Out[9]:
Combine public and private data to get more insights¶
In [10]:
print('e.g. # of Hong Kong v.s. Non Hong Kong studies (Social Science)')
print('(Just draft labeling! -- cite the figure at your own risk)')
Image('assets/hk-non-hk-studies-humanities.png')
Out[10]:
Major Steps of Scraping¶
- Download -- Get the raw materials (HTML/ PDF/ XLS)
- Parse -- Extract useful information from raw materials and put into structured format
Download -- A lot of command line utilities for quick hacks¶
In [11]:
%%sh
ls -1
In [12]:
%%sh
curl -s 'http://initiumlab.com/' | head -n 8
Download -- Human friendly in Python¶
In [13]:
import requests
html = requests.get('http://initiumlab.com/').content
html[:500]
Out[13]:
Parse¶
Not an easy task, generally:
- Need many trials and erros in practice.
- Could get something you don't want.
In [14]:
%%sh
curl -s 'http://initiumlab.com/' | grep title
In [15]:
%%sh
curl -s 'http://initiumlab.com/' | grep '<title'
Main problems in scraping¶
- Download
- Robot detection
- Authentication/ Authorisation
- Transfer error
- Encoding
- Incremental Crawling
- Get the right seed
- [Y] Scale-out
- Parse (focus on HTML)
- [Y] Find pattern
- [Y] Manual
- [Y] Machine learning
- [Y] Leverage pattern
- Deal with anomaly (e.g. broken page)
- [Y] Find pattern
[Y] items will be involved in this talk.
For mature project, you usually loop between Download and Parse, e.g. scrapy is a widely used framework.
Keywords of this demo:
- Quick
- Dirty Hack
HTTPie¶
Human-friendly command tool written in Python
In [16]:
%%sh
http get http://initiumlab.com | head -n 50 | tail -n 10
Command-line (text-interface; stdin/stdout) not good?¶
No. Nearly seamless integration:
In [17]:
lines = !http get http://initiumlab.com
lines[0:10]
Out[17]:
The parameters¶
':' HTTP headers:
Referer:http://httpie.org Cookie:foo=bar User-Agent:bacon/1.0
'==' URL parameters to be appended to the request URI:
search==httpie
'=' Data fields to be serialized into a JSON object (with --json, -j)
or form data (with --form, -f):
name=HTTPie language=Python description='CLI HTTP client'
':=' Non-string JSON data fields (only with --json, -j):
awesome:=true amount:=42 colors:='["red", "green", "blue"]'
'@' Form file fields (only with --form, -f):
cs@~/Documents/CV.pdf
'=@' A data field like '=', but takes a file path and embeds its content:
essay=@Documents/essay.txt
':=@' A raw JSON field like ':=', but takes a file path and embeds its content:
package:=@./package.json
You can use a backslash to escape a colliding separator in the field name:
field-name-with\:colon=valueTest requests¶
In [18]:
%%sh
http get 'http://httpbin.org/get' name==hupili at=='Scrape more with less codes!'
In [19]:
%%sh
http post 'http://httpbin.org/post' name==hupili at=='Scrape more with less codes!'
Caveats: --ignore-stdin is required in IPython notebook
Not a problem in command-line env.
Related issues: https://github.com/jkbrzt/httpie/issues/150
In [20]:
%%sh
http --form --ignore-stdin post 'http://httpbin.org/post' name==hupili at='Scrape more with less codes!'
Note: "Content-Type": "application/x-www-form-urlencoded; charset=utf-8",
In [21]:
%%sh
http --form --ignore-stdin post 'http://httpbin.org/post' name=hupili at='Scrape more with less codes!'
Note: "Content-Type": "application/json",
In [22]:
%%sh
http --ignore-stdin post 'http://httpbin.org/post' name=hupili at='Scrape more with less codes!'
Back to our Hong Kong RGC example¶
In [23]:
%%sh
http get https://cerg1.ugc.edu.hk/cergprod/scrrm00541.jsp | head -n 10
In [24]:
%%sh
http get https://cerg1.ugc.edu.hk/cergprod/scrrm00541.jsp | grep 155
Needs to find out why it doesn't give us the links
In [25]:
Image('assets/rgc-search-network-trace.png')
Out[25]:
Now use HTTPie to easily construct the query
In [26]:
%%sh
http post https://cerg1.ugc.edu.hk/cergprod/scrrm00541.jsp | grep 155
In [27]:
%%sh
http --ignore-stdin --form post 'https://cerg1.ugc.edu.hk/cergprod/scrrm00541.jsp' 'mode=search' 'sScheme=1' | grep 155
In [28]:
html_lines = !http --ignore-stdin --form post 'https://cerg1.ugc.edu.hk/cergprod/scrrm00541.jsp' 'mode=search' 'sScheme=1'
In [29]:
html_lines_with_a = list(filter(lambda l: '<A' in l, html_lines))
html_lines_with_a[:5]
Out[29]:
pQuery -- Grep for HTML¶
A wrap around pyQuery -- a Python library that allow you manipulate HTML in jQuery style.
Try plain grep first¶
In [30]:
%%sh
http --ignore-stdin --form post 'https://cerg1.ugc.edu.hk/cergprod/scrrm00541.jsp' 'mode=search' 'sScheme=1' \
| grep 155
In [31]:
%%sh
http --ignore-stdin --form post 'https://cerg1.ugc.edu.hk/cergprod/scrrm00541.jsp' 'mode=search' 'sScheme=1' \
| grep '<a'
Ignore case
In [32]:
%%sh
http --ignore-stdin --form post 'https://cerg1.ugc.edu.hk/cergprod/scrrm00541.jsp' 'mode=search' 'sScheme=1' \
| grep -i '<a' | head -n 5
In [33]:
%%sh
http --ignore-stdin --form post 'https://cerg1.ugc.edu.hk/cergprod/scrrm00541.jsp' 'mode=search' 'sScheme=1' \
| grep -i '<a' | grep -o 'HREF=".*"' | head -n 5
In [34]:
%%sh
http --ignore-stdin --form post 'https://cerg1.ugc.edu.hk/cergprod/scrrm00541.jsp' 'mode=search' 'sScheme=1' \
| grep -i '<a' | grep -o 'HREF=".*"' | cut -d'"' -f2 | head -n 5
Try pQuery¶
In [35]:
%%sh
http --ignore-stdin --form post 'https://cerg1.ugc.edu.hk/cergprod/scrrm00541.jsp' 'mode=search' 'sScheme=1' \
| pquery 'a' | head -n 5
In [36]:
%%sh
http --ignore-stdin --form post 'https://cerg1.ugc.edu.hk/cergprod/scrrm00541.jsp' 'mode=search' 'sScheme=1' \
| pquery 'a' -p href | head -n 5
In [37]:
%%sh
http --ignore-stdin --form post 'https://cerg1.ugc.edu.hk/cergprod/scrrm00541.jsp' 'mode=search' 'sScheme=1' \
| pquery 'a' -p href | wc -l
In [38]:
Image('assets/rgc-index-list.png')
Out[38]:
In [39]:
%%sh
http --ignore-stdin --form post 'https://cerg1.ugc.edu.hk/cergprod/scrrm00541.jsp' 'mode=search' 'sScheme=1' \
| pquery "table td[align='right'] a" -p href | wc -l
Further read: a case using HTTPie and pQuery¶
Scrape the info of 60 data science books and visualise their connection: http://www.kdnuggets.com/2015/09/free-data-science-books.html
- Demo repo: https://github.com/hupili/60-data-science-book-visualisation
- HTTPie: https://github.com/jkbrzt/httpie
- pquery: https://github.com/hupili/pquery (CLI wrapper of PyQuery)
In [40]:
%%sh
http --body 'http://www.kdnuggets.com/2015/09/free-data-science-books.html' |\
pquery '.three_ul li strong a' -f '"{text}",{href}' |\
head -n 8
In [41]:
Image('assets/data-science-books-graph.png')
Out[41]:
Scale-out -- all in command-line¶
Check the downloaded data¶
In [42]:
%%sh
http --ignore-stdin --form post 'https://cerg1.ugc.edu.hk/cergprod/scrrm00541.jsp' 'mode=search' 'sScheme=1' \
| pquery "table td[align='right'] a" -p href > path-list.txt
Next, let's download them all
In [43]:
%%sh
tail -n 1 path-list.txt | xargs -I{} http "https://cerg1.ugc.edu.hk/cergprod/{}" \
| pquery 'table.styleTableContent' -p html | head -n 10
In [44]:
%%sh
tail -n 1 path-list.txt | xargs -I{} http "https://cerg1.ugc.edu.hk/cergprod/{}" \
| pquery 'table.styleTableContent' -p html | pquery 'td' -p text
Use xargs -P for local multi-processing¶
In [45]:
%time page_lines = !tail -n 10 path-list.txt | xargs -I{} http "https://cerg1.ugc.edu.hk/cergprod/{}"
In [46]:
%time page_lines = !tail -n 10 path-list.txt | xargs -I{} -P5 http "https://cerg1.ugc.edu.hk/cergprod/{}"
Other quick parallel execution tools¶
A) My early dirty work: https://github.com/hupili/Lightweight-Distributing-Toolset
In Perl. 4 years ago. Do not use
B) GNU Parallel: http://www.gnu.org/software/parallel/
Written in Perl. Only need SSH access to remote (or local machine)
Cool, but...
C) PSSH: https://code.google.com/p/parallel-ssh/
- written in Python, good.
- Can brew install, good.
In [47]:
%%file hosts
localhost
In [48]:
%%sh
cat hosts
In [49]:
%%sh
pssh -h hosts -o output/ 'echo hello PSSH'
In [50]:
%%sh
ls output/
In [51]:
%%sh
cat output/localhost
Pandas¶
Easier for tabulared data.
In [ ]:
In [52]:
%%sh
tail -n 1 path-list.txt | xargs -I{} http "https://cerg1.ugc.edu.hk/cergprod/{}" \
| pquery 'table.styleTableContent' -p html | head -n 5
In [53]:
table_html = !tail -n 1 path-list.txt | xargs -I{} http "https://cerg1.ugc.edu.hk/cergprod/{}" | pquery 'table.styleTableContent' -p html
In [54]:
import pandas as pd
df_projects = pd.read_html('<table>%s</table>' % '\n'.join(table_html))
In [55]:
df_projects[0]
Out[55]:
Readability¶
We use a version ported to Python3:
https://github.com/hyperlinkapp/python-readability
(already included in the reuqirements.txt file)
In [56]:
from readability.readability import Document
import requests
html = requests.get('http://initiumlab.com/blog/20150922-jackathon3-review/').content
readable_article = Document(html).summary()
readable_title = Document(html).short_title()
In [57]:
print(readable_article[:1000])
In [58]:
HTML(readable_article[:1000])
Out[58]:
PyQuery¶
Let's fix the above URL problems
In [59]:
import pyquery
r = pyquery.PyQuery(readable_article)
r('p')
Out[59]:
In [60]:
r('video').attr('poster')
Out[60]:
In [61]:
r('video source').attr('src')
Out[61]:
In [62]:
r('video').attr('poster', 'http://initiumlab.com/%s' % r('video').attr('poster'))
Out[62]:
In [63]:
r('video').attr('poster')
Out[63]:
In [64]:
r('video source').attr('src', 'http://initiumlab.com/%s' % r('video source').attr('src'))
Out[64]:
In [65]:
r('video source').attr('src')
Out[65]:
In [66]:
r.html()[:1000]
Out[66]:
In [67]:
%%javascript
//IPython.OutputArea.auto_scroll_threshold = 9999;
IPython.OutputArea.prototype._should_scroll = function(){return false;}
In [68]:
HTML(r.html()[:1000])
Out[68]:
Scrapely¶
In [69]:
from scrapely import Scraper
s = Scraper()
In [70]:
help(s.train)
In [71]:
from urllib import parse
def get_localhost_url(url):
filename = parse.quote_plus(url)
fullpath = 'tmp/%s' % filename
html = requests.get(url).content
open(fullpath, 'wb').write(html)
return 'http://localhost:8888/files/%s?download=1' % parse.quote_plus(fullpath)
In [72]:
training_url = 'http://initiumlab.com/blog/20150916-legco-eng/'
training_data = {'title': 'Legco Matrix Brief (English)',
'author': 'Initium Lab',
'date': '2015-09-16'}
s.train(get_localhost_url(training_url), training_data)
In [73]:
testing_url = 'http://initiumlab.com/blog/20150901-data-journalism-for-the-blind/'
s.scrape(get_localhost_url(testing_url))
Out[73]:
In [74]:
testing_url = 'http://initiumlab.com/blog/20150922-jackathon3-review/'
s.scrape(get_localhost_url(testing_url))
Out[74]:
In [75]:
testing_url = 'http://initiumlab.com/blog/20151015-3d-infographic-user-testing/'
s.scrape(get_localhost_url(testing_url))
Out[75]:
In [76]:
blogs = !http get http://initiumlab.com/blog/ | pquery 'a.post-title-link' -p href
blogs
Out[76]:
In [77]:
infos = []
for b in blogs:
infos.extend(s.scrape(get_localhost_url('http://initiumlab.com/blog/' + b)))
In [78]:
infos
Out[78]:
In [79]:
import pandas as pd
df_blogs = pd.DataFrame(infos)
df_blogs['title'] = df_blogs['title'].apply(lambda x: x[0].strip())
df_blogs
Out[79]:
Summary¶
Theme: scrape more with less codes
Keywords: quick and dirty hacks
Environment:
- Mostly command-line
- Some in Python shell
Human friendly HTTP interface:
- CLI:
HTTPie - Python REPL:
requests
Scale-out
- CLI:
- Single machine:
xargs -P - Multiple machine:
pssh
- Single machine:
- Python REPL:
- Better to wrap as individual scripts and do multi process
Manual parse:
- CLI:
pQuery - Python REPL:
pyQueryfor FE peoplepandasuseful for tabulared data
Automatic parse, in Python REPL:
PyReadability: the main body of a pagescraply: learn patterns from your labelling
In [ ]: